Sources of Variability and Adaptive Tasks

It is known that micro-task worker performance fluctuates. While between-worker variability has been studied and has been used to define filters (e.g., to filter out “bad” workers), within-worker variability (i.e. how each worker's performance varies over time) has received less attention. Better understanding of the sources of such variability will result in the design of better filters, and more importantly, can inspire the development of adaptive tasks. In an adaptive task, between-worker variability is reduced by adapting the type and difficulty of a job to a worker, while within-worker variability is addressed by reacting via feedback to a change in worker performance. This paper presents evidence of within-worker variability on Amazon Mechanical Turk, and defines a set of sources of variability and describes how adaptive tasks could be designed to attend to them. Author